Scene change detection is an important problem in video systems. In a typical process for editing a video production, an operator uses a written log which describes the contents of each scene in the raw video footage and associates that information with the location on a tape using a time code number. An operator manually produces the log by viewing the tape and making notations when a scene change occurs.
Some prior art video systems overcome the time-consuming and cumbersome process of manual scene change detection using pixel-based scene change detection. In these pixel-based systems, later frames of a video are compared against earlier frames and the video is automatically segmented into scenes based on differences between the pixels in adjacent frames.
Computing devices, such as desktop computers, laptops, and handhelds, ostensibly include video systems that provide video information to end users. For example, most users interact with software programs through a graphical user interface that is presented on a computer monitor. The output of these video systems can be recorded and manipulated like any other traditional video data, such as a movie or television program.
The application of prior art techniques for video processing to video data recorded from a computer environment, however, only accounts for visual changes in the recorded data and ignores the semantics of changes to the computing environment reflected in the pixels changing in the recorded data. In essence, prior art systems may be able to observe that “something” happened in the recorded data, but these systems are not able to understand “what” happened in the computing environment that resulted in the changes in the recorded data, or understand the actions associated with or the events signified by that happening.
For example, some known systems utilize scene change detection to segment video into chapters. This technique, however, is typically limited to post-production analysis and is used to create chapters when there is a significant pixel difference between frames. This scene change detection feature is not designed to track events and changes occurring in video data recorded in a computing environment and cannot automatically identify portions of the recorded data associated with events or changes. Accordingly, known systems do not easily allow for the identification or playback of video data recorded from a computing environment associated with events that occurred while the video data was recorded, such as an application starting, a change in the foreground application, or an application going into full-screen mode.
There exists, therefore, a need for methods and apparatus for automatically associating video data recorded from a computing environment with other meaningful information, such as information describing changes in a computing environment.